Increasing the step of the Newtonian Decomposition Method for Support Vector Machines

The Newtonian method is a standard iterative method with quadratic convergence rates for solving large optimization problems. The Support Vector Machine problem formulation requires the solution of large datasets, which is ideal for the application of Newtonian methods. We point out that the Sequential Minimal Optimization (SMO) algorithm is a Newtonian Decomposition Method that is popular due to its efficiency but unfortunately does not scale too well with large data sets. The algorithm is an implementation of the decomposition method, which solves a sequence of sub problems instead of the entire problem at once. In this report, we introduce an extrapolation parameter to the SMO update method and investigate the effect on the rate of convergence of this algorithm. We first show that the SMO update method is Newtonian and that extrapolation ensures the update is norm reducing on the objective function. We also investigate the bounds of extrapolation and derive optimal estimates for this parameter. It was observed that choosing the working set pair according to some partial order does result in slightly faster speedups in algorithm performance.